🎉 Destination Oracle - Added support for connection via ssh tunnel #6370

VitaliiMaltsev · 2021-09-22T09:34:04Z

What

The goal of this PR is to add SSH only for the Oracle Destination. It does not try to deal with how it can be reused for other sources or destinations.

How

Handles injecting the SSH tunnel in the spec, check, getConsumer methods of Oracle Destination directly. This makes it relatively straightforward to inject without worrying about touching other jdbc dbs.

Tested with AWS EC2 bastion and RDS Oracle:

or

der

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

sherifnada

LGTM but can you use the container approach?

sherifnada · 2021-09-24T16:41:02Z

.github/workflows/publish-command.yml

@@ -130,6 +130,8 @@ jobs:
          MIXPANEL_INTEGRATION_TEST_CREDS: ${{ secrets.MIXPANEL_INTEGRATION_TEST_CREDS }}
          MSSQL_RDS_TEST_CREDS: ${{ secrets.MSSQL_RDS_TEST_CREDS }}
          PAYPAL_TRANSACTION_CREDS: ${{ secrets.SOURCE_PAYPAL_TRANSACTION_CREDS }}
+          ORACLE_SSH_KEY_TEST_CREDS: ${{ secrets.ORACLE_SSH_KEY_TEST_CREDS }}


can we instead use the approach you took in #5811 ?

sure, i'll modify on container approach once #6312 will be merged

@sherifnada I changed the integration tests for the bastion container approach

@sherifnada also how do you think maybe we need to create separate tasks to also change the integration tests for Oracle Source, MySQL Source and Destination, MSSQL Source and Destination with docker bastion container?

@VitaliiMaltsev yup let's create issues on the backlog to do this, and make sure with the team that any new SSH implementations follow the bastion approach

cc @alexandr-shegeda

# Conflicts: # tools/bin/ci_credentials.sh

CLAassistant · 2021-09-27T17:17:28Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

vmaltsev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…rets for Oracle db

VitaliiMaltsev · 2021-09-28T10:49:15Z

/publish connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1282221745
❌ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1282221745

VitaliiMaltsev · 2021-09-28T10:57:48Z

/publish connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1282245458
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1282245458

vmaltsev added 2 commits September 21, 2021 18:07

Oracle destination with ssh tunneling

1a6ea4d

add ssh key integration testing for Oracle destination

3783c88

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 22, 2021

VitaliiMaltsev temporarily deployed to more-secrets September 22, 2021 09:34 Inactive

fix checkstyle

75f0aa7

VitaliiMaltsev temporarily deployed to more-secrets September 22, 2021 10:26 Inactive

VitaliiMaltsev changed the title ~~Vmaltsev/5969 oracle destination ssh tunnel~~ tada Destination Oracle - Added support for connection via ssh tunnel Sep 22, 2021

VitaliiMaltsev temporarily deployed to more-secrets September 22, 2021 11:55 Inactive

VitaliiMaltsev changed the title ~~tada Destination Oracle - Added support for connection via ssh tunnel~~ Destination Oracle - Added support for connection via ssh tunnel Sep 22, 2021

VitaliiMaltsev changed the title ~~Destination Oracle - Added support for connection via ssh tunnel~~ tadaDestination Oracle - Added support for connection via ssh tunnel Sep 22, 2021

VitaliiMaltsev changed the title ~~tadaDestination Oracle - Added support for connection via ssh tunnel~~ 🎉 Destination Oracle - Added support for connection via ssh tunnel Sep 22, 2021

VitaliiMaltsev requested review from alexandr-shegeda and DoNotPanicUA September 23, 2021 09:49

VitaliiMaltsev linked an issue Sep 23, 2021 that may be closed by this pull request

Add support & test cases for Oracle Destination via SSH tunnels #5969

Closed

alexandr-shegeda approved these changes Sep 24, 2021

View reviewed changes

VitaliiMaltsev requested review from etsybaev and removed request for DoNotPanicUA September 24, 2021 10:17

VitaliiMaltsev marked this pull request as ready for review September 24, 2021 10:29

VitaliiMaltsev requested a review from sherifnada September 24, 2021 10:29

sherifnada approved these changes Sep 24, 2021

View reviewed changes

vmaltsev added 3 commits September 27, 2021 11:30

Merge branch 'master' into vmaltsev/5969-oracle-destination-ssh-tunnel

98a3f26

# Conflicts: # tools/bin/ci_credentials.sh

add container approach to Oracle Destination Integration Tests

0a61b3c

add container approach to Oracle Destination Integration Tests

9870d24

VitaliiMaltsev temporarily deployed to more-secrets September 27, 2021 17:18 Inactive

add container approach to Oracle Source Integration Tests/ delete sec…

6314486

…rets for Oracle db

VitaliiMaltsev temporarily deployed to more-secrets September 28, 2021 08:40 Inactive

Oracle Destination - bump version to 0.1.8

36deae3

VitaliiMaltsev temporarily deployed to more-secrets September 28, 2021 10:49 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 10:51 Inactive

remove oracle ssh secrets from ci_credentials.sh

36bd14a

VitaliiMaltsev temporarily deployed to more-secrets September 28, 2021 10:58 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 11:00 Inactive

VitaliiMaltsev merged commit c256114 into master Sep 28, 2021

VitaliiMaltsev deleted the vmaltsev/5969-oracle-destination-ssh-tunnel branch September 28, 2021 11:25

This was referenced Sep 29, 2021

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6532

Closed

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6539

Closed

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6548

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination Oracle - Added support for connection via ssh tunnel #6370

🎉 Destination Oracle - Added support for connection via ssh tunnel #6370

VitaliiMaltsev commented Sep 22, 2021 •

edited

Loading

sherifnada left a comment

sherifnada Sep 24, 2021

VitaliiMaltsev Sep 24, 2021

VitaliiMaltsev Sep 27, 2021

VitaliiMaltsev Sep 27, 2021

sherifnada Sep 27, 2021

CLAassistant commented Sep 27, 2021

VitaliiMaltsev commented Sep 28, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 28, 2021 •

edited by github-actions bot

Loading

🎉 Destination Oracle - Added support for connection via ssh tunnel #6370

🎉 Destination Oracle - Added support for connection via ssh tunnel #6370

Conversation

VitaliiMaltsev commented Sep 22, 2021 • edited Loading

What

How

Recommended reading

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Sep 24, 2021

Choose a reason for hiding this comment

VitaliiMaltsev Sep 24, 2021

Choose a reason for hiding this comment

VitaliiMaltsev Sep 27, 2021

Choose a reason for hiding this comment

VitaliiMaltsev Sep 27, 2021

Choose a reason for hiding this comment

sherifnada Sep 27, 2021

Choose a reason for hiding this comment

CLAassistant commented Sep 27, 2021

VitaliiMaltsev commented Sep 28, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 28, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 22, 2021 •

edited

Loading

VitaliiMaltsev commented Sep 28, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 28, 2021 •

edited by github-actions bot

Loading